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SUMMARY 

Increasing people’s ability to solve complex problems is more and more often 
being seen as an integral part of vocational education. While there have been 
numerous empirically-based approaches to the didactic structuring of teaching 
and learning arrangements by which students’ ability to solve problems can be 
increased, knowledge of how to evaluate a person’s ability to solve problems is 
far more limited. There is a lack of testing instruments that are inexpensive to 
implement and evaluate and take account of the features of ill-defined problems 
as they occur in professional practice. This article describes, with reference to 
a pilot study, the further development of a method of measuring a person’s abil- 
ity to solve problems (AIT according to Sembill) which is reliable but too expen- 
sive to evaluate in practice. The new MAPS instrument (Measurement and As- 
sessment of Problem Solving) is more structured in terms of both performance 
and evaluation. The results of the pilot study indicate a high level of reliability 
and validity of the new instrument, these results having to be confirmed in a fol- 
low-up study, the design of which is also presented in this article. 
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Problem 

Young people in vocational education and those entering the job mar- 
ket nowadays face a rapidly changing, increasingly complex world. This re- 
structuring in the business world j 1 ) (cf. Buttler, 1992; Schunck, 1993, 
Dohmen, 1999; Reetz, 1999; Achtenhagen, 2000; Picot, 2000; Kessler, 2003), 
often referred to simply as ‘megatrends’, also leads to a change in require- 
ments for business staff (white collar jobs). In particular, the question is raised 
as to the extent to which knowledge acquired during education is retained 
over time and whether other, higher cognitive abilities are not also crucial 
when it comes to carrying out the work and tackling problems in one’s pro- 
fessional life. A person’s ability to solve problems is discussed as being a 
key skill and may be defined as a knowledge of how to deal adequately with 
complex, not fully understood and ever-changing realities (cf. Sembill, 1992a; 
Bransford; Stein, 1993; Sembill, 1995; Wuttke, 1999; Wolf, 2003). 

Increasing a person’s ability to solve problems should therefore already 
form an integral part of one's professional training: ‘Learning to solve prob- 
lems is the most important skill that students can learn in any setting. In pro- 
fessional contexts, people are paid to solve problems, not to complete ex- 
ams.’ (Jonassen, 2004, XXI). However, traditional teaching in vocational 
colleges has hitherto failed to promote this in any systematic way. This is 
particularly evident when students are supposed to apply acquired knowl- 
edge in problem situations but are unable to do so ( 2 ). Failures to apply knowl- 
edge are attributable not least to the teaching practice still prevailing in many 
(vocational) schools. For the most part (over 70% of teaching time), the 
teacher presents facts and details. There is barely any time left to look at 
the application of knowledge and any particular increasing of a person’s abil- 
ity to solve problems (Hage; Bischoff; Dichanz; Eubel; Oehlschlager; 
Schwittmann, 1985, Dichanz; Schwittmann, 1986; Bohl, 2000; Patzold; Klus- 
meyer; Wingels; Lang, 2003). Although teachers generally identify a per- 
son’s ability to solve problems as being a skill that is crucial to success in 
the workplace, they do almost nothing to help their students acquire this 
skill. There are three possible reasons for this: 

Teachers think that rudiments and basic skills have to be taught first be- 
fore they can move on to more demanding subjects like problem-solving. 
As there are plenty of rudiments to teach in any subject and, furthermore, 
increasing a person's ability to solve problems takes time, they tend to put 
it off from one year to the next. 


C) Internationalisation of markets, globalisation of the use of resources, introduction of new tech- 
nologies, breakdown of traditional values, development into a service society, demand for 
highly qualified staff. 

( 2 ) The problem is far from new. Back in 1929, Whitehead referred in this respect to the prob- 
lem of inert knowledge. Inert knowledge is knowledge that can be repeated by people if they 
are explicitly asked for it, but cannot be used and applied spontaneously in situations where 
problems have to be solved. 
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Most teachers are unsure how learning environments can be structured 
in order to increase a person’s ability to solve complex problems (lack of 
known methods). 

While a person's ability to solve problems is established as a learning 
objective in curricula, it is unclear how it can be evaluated. Standardised 
tests in schools and as part of one’s vocational training focus on the repro- 
duction of facts, not on a person’s ability to solve ill-defined problems. As 
learning processes are largely influenced by the tests that follow, teachers 
and students still focus on acquiring and reproducing factual knowledge. 

In the light of these deficiencies, our research concentrates on the key 
areas of the structuring and implementation of learning environments to in- 
crease a person’s ability to solve problems, and measurement of the ac- 
quired ability to solve problems. The following comments focus on the prob- 
lems of measurement from the point of view of the structuring and imple- 
mentation of learning environments (cf. Sembill, 1992a, Wuttke, 1999, Schu- 
macher, 2002, Wolf, 2003 and Seifried, 2004). 


Learning environments for increasing 
a person’s ability to solve problems 

There are various approaches defining the ‘ideal typical’ steps to be tak- 
en in order to solve a problem and therefore giving clues as to how learn- 
ing environments should be structured to increase a person’s ability to solve 
problems. Bransford and Stein (1993) propose, for example, the IDEAL frame- 
work (Identify, Define, Explore, Anticipate and Act, Look and Learn). In the 
German-speaking sphere, Sembill (1992a) has developed a comparable 
framework model that he describes as an analytical ideal type of planned 
actions (see Fig. 1). 

The characteristic components of complex problem-solving processes 
contained therein are identified in problem-solving research and cognitive 
psychology (cf. Dorner, 1976, Dorner, 1983, Sembill, 1992a, Dorner, 1999). 
They form the basis for structuring complex learning environments in which 
students: 

(1) Identify the problem by: 

(1.1) Analysing and assessing a given situation; 

(1 .2) Setting objectives; 

(1 .3) Identifying possible discrepancies between the initial situation and 
the desired objectives; 

(2) Use their existing knowledge; 

(3) Gather the necessary information; 

(4) Propose solutions (action or problem-solving programme); 

(5) Analyse side effects in relation to the desired effects; 

(6) Implement proposed solutions; 

(7) Check and evaluate the result of the solution to the problem and 

(8) Improve future problem-solving processes. 
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Figure 1. The Analytical Ideal Type (AIT) of planned Action (Sembill, 1992a; 
Sembill; Wolf; Wuttke; Schumacher, 2002) ( 3 ). 



Most of these elements are also found in what are referred to as con- 
structivistically-based learning environments like anchored instruction 
(CTGV, 1 992), intentional learning (Bereiter & Scardemelia, 1 989) and learn- 
ing with cognitive tools (Kommers; Jonassen; Mayes, 1992). 

The self-organised learning developed in our research group is based 
on the basic principle described above and therefore enables a person’s 
ability to solve complex problems to be increased (cf. Sembill, 1992a; 
Wuttke, 1 999; Sembill; Wolf, 2000). Self-organised learning can be described 


( 3 ) This plan is an elaborated form of the TOTE unit (Test-Operate-Test-Exit; Miller; Galanter; 
Pribram, 1 960), which is regarded as the unit on which problem-solving processes are based. 
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in terms of four main characteristics (cf. Sembill; Wolf; Wuttke; Schumacher, 
2002; Wolf, 2003): 

• Central to self-organised learning are the problem-solving activities of 
students. The problems to be tackled are complex and are generally 
solved in a project-based and group-oriented environment. 

• The planning, implementation and assessment of learning processes 
is - as far as possible - left to the students ( 4 ). Self-organised learning 
also has to cover the definition of and reflection on objectives and the 
assessment of and reflection on a person’s own actions and his solu- 
tions to the problem (cf., in this respect, the ‘ideal typical’ steps for solv- 
ing a problem shown in Fig. 1). 

• In self-organised learning, each person naturally also does his own learn- 
ing. However, in addition, learning for others (sharing work when tack- 
ling problems and presenting results) and with others are central ele- 
ments in the design of self-organised learning. It is also assumed here 
that arguments within groups and with the teacher and the expression 
of people’s own ideas promote reflection and analysis and help the gen- 
eration of knowledge and the ability to solve problems (cf. Wuttke, 2005). 

• When solving complex, realistic problems, there is always the risk that 
mistakes will be made and first attempts will fail. However, it also means 
that people can learn from their mistakes, reach an understanding in- 
dependently and build up skills (cf. Spychiger; Gut; Rohrbach; Oser, 
1999; Oser; Spychiger; Mahler; Reber, 2002; Spychiger, 2003). 

Self-organised learning has already been introduced and evaluated 
in five vocational colleges. In two of the studies, a virtual business was 
also created as an ‘anchor’ to help in problem-solving (a furniture com- 
pany with imaginary data such as price lists, catalogues, calculations, lists 
of materials, delivery dates, etc.). Within the framework of the respective 
teaching units, problems relating to established curriculum subjects had 
to be solved. For example, students were put in the shoes of a furniture 
manufacturer when dealing with the subject of ‘Materials business’. This 
manufacturer, at the request of a travel agent, has to submit a tender for 
the furnishing of an entire new branch to be opened in a student district. 
Computer equipment - apart from sector-specific software - also has to 
be provided. While tackling this problem (over the course of about 20 hours 
of teaching), students have to solve a large number of subsidiary prob- 
lems. For example, decisions have to be made as to whether a new line 
of furniture will be purchased or made by the company itself or whether 
a just-in-time delivery is less or more expensive than keeping the furni- 
ture in stock. 

A summary of the main findings of the studies so far shows that, when 
following a course of self-organised learning and in comparison with con- 
ventionally taught lessons, students: 


( 4 ) This all of course takes place within the framework of prescribed curricular requirements. 
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• are able to master the subject matter just as well as students who are 
taught it by teachers. In tests based on learning objectives, they come 
off just as well as those in the control group despite poorer initial qual- 
ifications in some cases; 

• show a far greater ability to solve problems after completing the respec- 
tive teaching unit. This applies both to problems specific to the subject 
and to problems from other spheres of the lives of young people not cov- 
ered in the subject; 

• show a greater degree of self-motivation required for analysis-oriented 
learning (intrinsic motivation and interest); 

• broaden their range of learning strategies and apply learning strategies 
more appropriately; 

• get fully involved in their respective learning groups and classes and 
feel that they are being taken seriously ( 5 ). 

If the first finding were now to be looked at in isolation - 'self-organised 
learning achieves the same level of knowledge as conventional teaching’ 
- one would have to wonder whether the undoubtedly higher cost of self- 
organised learning is actually worth it. However, as explained above, the 
aim is not just to enable students to repeat factual knowledge, it is also to 
give them the ability to solve problems as is required when meeting the chal- 
lenges of the workplace. But in order to be able to comment on whether a 
person’s ability to solve problems has actually increased, and in order to 
be able to include this skill in the measurement of performance, a person's 
ability to solve problems has to be made measurable. 


Measuring a person’s ability to solve complex 
problems 

In psychological research and research into learning and teaching, in- 
formation provided by students themselves are generally gathered by means 
of questionnaires in order to measure their ability to solve problems (e.g. 
Staudel’s Skills Questionnaire, 1987 or Dirksmeier’s Diagnostic Inventory 
of Problem-Solving DIP, 1 991 ). However, this does not tell us how well stu- 
dents are actually able to solve problems in their professional environment 
( 6 ). In our work we therefore concentrate on measuring actual performance 
in problem-solving. Students are given complex and realistic written prob- 
lems to be solved by giving written answers. For this purpose, Sembill (1 992a„ 
1 992b) developed a system to enable the quality of written solutions to prob- 
lems be assessed. It includes both quantitative and qualitative criteria and 


( 5 ) Detailed descriptions and discussions of the findings are given, for example, in Wuttke, 1 999; 
Wolf, 2003; Santjer-Schnabel, 2002, Sembill, 2004, Seifried, 2004; Schumacher, 2002). 

( 6 ) A strongly subject-oriented Test of ability to solve sector-specific problems’ is described by 
Hussy; Seeling (2004a, 2004b). 
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Figure 2. Example of an ill-defined problem 


Please imagine you are faced with the following situation: 

You are Head of the Personnel Department at a furniture factory (Justus-Liebig-Buromobelwerke, JLB) in GieBen, 
South Hesse. The company has just been taken over by the Swedish HAVARTI Group. 

The Personnel Manager Mrs Olsen wants Mrs Mertens - an office worker in the Personnel Department’s pay- 
roll office - to be trained in PowerPoint so that she can produce transparencies for Mrs Olsen’s advertising 
presentations. Mrs Mertens refuses because it is not part of her job description. ‘I already have my hands full 
with the wages accounting’, she explains. 

Mrs Olsen is annoyed and consults you in your capacity as her manager. ‘I think job descriptions are a waste 
of time. What good do they do anyway? They only make staff inflexible. And it takes ages to produce these 
descriptions. The business we are in is constantly changing and the requirements of our staff change at the 
same time’. Mrs Olsen asks you to send out a memo stating that job descriptions are no longer binding. 

Please imagine you are the Manager and try to resolve the dispute. Think of the possible consequences of do- 
ing away with binding job descriptions. 

Please describe how you weigh up the arguments for and against various solutions and which solution you 
go for. Remember to give your reasons. 

You have 30 minutes. 


contains the ideal steps in a problem-solving process (see Fig. 1 ; cf. also 
Dorner, 1976; Dorner; Kreuzig; Reither; Staudel, 1983): 

• Analysis of the initial situation 

• Setting of objectives 

• Developing problem-solving strategies or measures 

• Controls to check the appropriateness of the solution to the problem (tak- 
ing account of main and side effects). 

Students are given descriptions of problems (see, for example, Fig. 2) 
and are supposed to provide written solutions to them. 

The written solutions to these sorts of problems are assessed in a first 
step according to quantitative criteria. Analysis is carried out here as to 
whether the students have carried out all problem-solving steps appropri- 
ately (for the system, see Fig. 3). 
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Figure 3. Overview of the evaluation categories for assessing 
problem-solving ability 



Below is a brief overview of how the evaluation is carried out. Detailed 

instructions for codes used are given by Sembill (1 992b) and Wuttke (1 999). 

(1) Coding for the category ‘Analysis of the initial situation' 

(actual situation): 

AS1 : Number of facts having a bearing on the problem. This category pro- 
vides information on the student’s ability to obtain relevant informa- 
tion (from texts). 

AS2: Comments, opinions and assessment of these facts (critical analy- 
sis of available information); 

AS3: Number of people/groups specified in the problem having different in- 
terests (who is relevant to the initial situation, who has to be taken into 
account). 


Coding example: 


Example of a student’s solution 

AS1 

AS2 

AS3 

Mrs Mertens is an office worker in the payroll office, so she 
possibly knows nothing about advertising. 

1 

1 

1 

1 think it is a good idea to ignore the job description. 


1 


1 think Mrs Mertens is right to refuse to produce the trans- 
parencies. 


1 

(1) 
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(2) Coding for the category ‘Setting objectives’ (desired situation) 

DS1 : Number of objectives specified; 

DS2: Production of a hierarchy of objectives (importance, sequence, pri- 
orities). 


Coding example: 


Example of a student’s solution 

DS1 

DS 2 

If Mrs Olsen needs help, she should persuade her manager to em- 
ploy an assistant for her. 

1 


At the moment, it is probably more important to do the work than 
to waste time in a power struggle. 

2 

1 


(3) Coding for the category ‘Measures’ 

Ml : Number of measures proposed; 

M2: Giving reasons for the measures; 

M3: Weighing up the measures (likelihood of success, main effects and 
side effects); 

M4: Decision on measures to be taken. 


Coding example: 


Example of a student's solution ( 7 ) 

Ml 

M2 

M3 

M4 

You should first talk about their differences. 

1 




They should share the work and do the transparencies to- 
gether. 

1 




If they do it together, both can manage their work. 


1 



It is better to reach a compromise than to impose rules (this 
is directed at Mrs Mertens, who is relying on her job descrip- 
tion and refusing to produce the transparencies). 

2 


1 


1 think it would be best if there were job descriptions but the 
managers also asked employees to be flexible. 




1 


(4) Coding for the category of ‘Controls’ 

Cl : Facts in the solution are referred back to the initial situation; 

C2: Description of the consequences of the solution for the initial situa- 
tion; 

C3: Facts in the solution are referred to the desired situation; 

C4: Description of the consequences of the solution for the desired situ- 
ation; 

C5: Facts in the solution are referred to the measures; 

C6: Description of the consequences of the measures. 


( 7 ) In the evaluation, it is striking that almost all participants specify measures but hardly ever 
give reasons. 
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Coding example: 


Example of a student’s solution 

Cl 

C2 

C3 

C4 

C5 

C6 

1 would explain to Mrs Olsen that it would be complete 
chaos if there were no compulsory job descriptions. 

1 

1 





It might be useful to mention in future job adver- 
tisements that employees have to be flexible and 
cooperative, even if they are asked to carry out 
tasks that do not precisely come under their job 
description. 



1 




Employees would then be unlikely to insist on 
doing only what comes under their job des- 
cription. 




1 



If he could persuade the two women to cooperate 
and do the work together, it would be much better 
for the workplace atmosphere. 





1 

1 


The categories can then be added together to give an overall score, the 
‘Analytical Ideal Type’. A high score indicates a highly developed ability to 
solve problems. 

In the next step, the quality of the problem-solving is analysed by ex- 
pert rating. The categories to be taken into account here are: 

• Use of declarative and procedural knowledge (giving reasons and al- 
ternatives, logical comprehensibility and prospect of success of the prob- 
lem-solving); 

• Complexity of the problem-solving and the reasons given (quality of the 
theoretical model); 

• Formulation of original objectives and ideas (e.g. moral considerations). 

In previous studies, this evaluation system has given results that were 
consistent both with the assessments of teachers and with our own assess- 
ments and observations in the classroom (cf. e.g. Wuttke, 1999; Wolf, 2003; 
Seifried, 2004). The problem is the large amount of time required to eval- 
uate written problem-solving, making the instrument unsuitable for use in 
schools. As there are a number of evaluation steps to go through, sever- 
al lessons have to be set aside for each solution. Furthermore, the results 
are hugely dependent on how motivated and prepared students are to write 
down what they think. Account also has to be taken of the fact that, because 
the questions are very open, students will not mention certain aspects that 
they do in fact know. 

As a result, the instrument is currently being developed to produce a more 
practical instrument for measuring a person’s ability to solve problems (MAPS 
- Measurement and Assessment of Problem Solving Skills). This instrument 
cuts down the time spent on evaluation and should be less dependent on 
the student’s motivation and ability to put thoughts into writing. MAPS com- 
prises a combined analysis of quantitative and qualitative aspects of prob- 
lem-solving. Unlike the original AIT method, students are also given specif- 
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ic questions along with the problem (of. Fig. 4). These should structure and 
help in the solutions and answers and prompt answers relating to all aspects, 
even for students who are not so motivated and/or are unable to put their 
answers in writing. Together with an evaluation sheet, the questions can help 
teachers ascertain the students' ability to solve problems. 


Figure 4. Problem together with specific questions 

Problem for the students 

Please imagine you are faced with the following situation: 

... the problem already described in Fig. 2 is inserted here 
Questions: 

Initial situation: 

• Please describe the situation in your own words. 

• What information is missing? Who would you ask for it, who could help you? 
Where would you look for additional information? 

• Please describe the assumptions you are making with respect to missing in- 
formation (when tackling this problem, you can work on the basis of assumptions 
because you currently have no access to additional information). 

Objectives: 

• Which objectives would be conceivable in relation to the situation described 
above? 

• Which would you try to achieve and why? 

Measures: 

• What measures could help you to achieve the objectives? 

• Please say why you think these measures are appropriate. 

• Are some measures better than others? Please take account not only of the ob- 
jectives you are trying to achieve, but also of possible (undesired) side effects. 

• Decide which measure(s) is/are most appropriate. 

(Theoretical) control: 

• How do your solutions alter the initial situation? 

• How successful is your solution given the objective you are trying to achieve? 

• How promising is/are your measure(s)? 
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The students’ answers are evaluated using the following assessment 
system: 


Figure 5. Evaluation sheet for solutions to problems 


MAPS assessment sheet 

Analysis of the initial situation 

Student gives a comprehensive description of the sit- 
uation. 

Yes (in own words) - to a certain 
extent - no 

The student’s description covers the main points. 

Completely - to a certain extent - 
not at all 

Student indicates missing/required information. 

Entirely sufficiently - not much - 
none 

Student formulates ideas as to where/from whom 
missing information can be found. 

Many - some - none 

Student formulates assumptions on missing informa- 
tion. 

Many - some - none 

The student’s assumptions are reasonable. 

All - some - none 

Definition of objectives 

Student formulates objectives. 

Many - some - none 

The objectives are reasonable. 

All - some - none 

Student formulates objectives for all those involved 
in the problem. 

For all - only for some - for none 

Student chooses one or more objectives. 

Several objectives - one objective 
- no objectives 

Student says why s/he chose the objective(s). 

Yes, explicitly - yes, implicitly - no 

The choice of objective(s) is well-founded. 

Yes - to a certain extent - no rea- 
sons or poor reasons 

Measures and action plans 

Student proposes various strategies/action plans. 

Yes -only a few -none 

Student explains why s/he thinks these measures are 
reasonable. 

Yes - partly - no 

Student weighs up various measures against one an- 
other. 

Yes - partly - no 

Student takes account of possible (undesired) side 
effects. 

Yes - partly - no 

Student decides which measure(s) is/are reasonable. 

Yes - no 

Control 

Student analyses whether and how measures might 
alter the initial situation. 

Yes - to a certain extent - no 

Student analyses whether the measure(s) is/are 
successful in achieving the desired objective(s). 

Yes - to a certain extent - no 

Student analyses whether the measure(s) is/are rea- 
sonable. 

Yes - to a certain extent - no 
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For each answer, marks are awarded according to the level shown (from 
0 to 2). In this way, performance in the individual sections and an overall 
score for a person’s ability to solve problems can be ascertained. The eval- 
uation table is in principle suitable for all subject areas ( 8 ), but subject ex- 
perts are required to carry out the evaluation in order to enable assessment 
of the students’ performance. 


MAPS evaluation: First findings from 
a pilot study 

Method 

For MAPS evaluation, a first pilot study with students has been carried 
out so far; the second test is to start in schools in autumn 2005. In the pi- 
lot study, nine students whose main course was business education ( 9 ) were 
looked at and given two similar problems. Both related in terms of their sub- 
ject to previously attended business education seminars. One problem was 
open (like the one in Fig. 2), the second problem was, as shown in Fig. 4, 
supplemented by specific questions. The first problem was assessed by two 
encoders according to the AIT system, the second - also by two encoders 
- according to the MAPS evaluation system. The results of this study should 
provide starting points when it comes to answering the following research 
questions ( 10 ). 

How reliable are the measurements of the two AIT and MAPS instru- 
ments? The interrater reliability (Cohen’s Kappa) is used check this. 

The results relating to a person’s ability to solve problems achieved so 
far using the AIT system are satisfactory insofar as they confirm theoreti- 
cally expected findings. For example, there is, in accordance with the the- 
ory, a positive correlation between analysis and a person’s ability to solve 
problems. Such findings indicate that, using the AIT system, a valid instru- 


ct The two instruments (AIT and MAPS) have so far only been trialled in German vocational 
colleges and universities. However, as they are independent of any subject, they could also 
be used for measurement purposes in other European and non-European educational es- 
tabishments. The only requireement is a subject expert to assess the quality of the solutions. 

( 9 ) The participants in the sample are therefore future teachers at vocational colleges in Ger- 
many. The sample is unrepresentative for numerous reasons. To begin with, it is too small 
to enable the findings to be generalised. Furthermore, MAPS is being developed for use in 
vocational colleges, in other words for a target group which is a) younger and b) usually less 
well educated than the sample in the pilot study. For these reasons, other studies of the pop- 
ulation for which the instrument is designed are required. Design considerations in this re- 
spect are presented in the course of the paper. 

( 10 ) As already mentioned, the sample of 9 participants is extremely small and the findings are 
certainly unrepresentative. As a result, the follow-up study will be carried out on a much broad- 
er basis. 
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merit for measuring a person's ability to solve problems could exist ( 11 ). It 
should now be examined whether MAPS also measures a person's abili- 
ty to solve problems. To do this, the relationship between the two meas- 
ures of a person’s ability to solve problems (measured by the AIT system 
and measured by the MAPS system) is determined. Ranking correlations 
are calculated with respect to the subdivisions of a person’s ability to solve 
problems (actual situation, objectives, measures and control) and with re- 
spect to the overall score (AIT). 

Findings of the pilot study 

When calculating the interrater reliability, findings were as expected. As 
both the tackling of the problem and the evaluation are more structured and 
prescribed with MAPS, the measures of the participants' ability to solve prob- 
lems correlate much more often than with AIT evaluation. The interrater re- 
liability within the framework of the AIT evaluation can only be assessed 
as being satisfactory (Cohen’s Kappa = 0.66). Correlation within the frame- 
work of the MAPS evaluation, on the other hand, is very acceptable with 
a Kappa of 0.89. 

The ranking correlations between the two measuring instruments indi- 
cate that both instruments at least measure similar constructs (Fig. 6). 

The correlation coefficients generally indicate a medium to high corre- 
lation between the two instruments. The statistical validation (significance) 
that is not available in all cases is presumably attributable to the small sam- 
ple. Nevertheless, the findings can be regarded as being an indication that 
the instruments measure similar things. 


Figure 6. Ranking correlations between AIT and MAPS findings in 
unrepresentative testing (N = 9) 



AIT and MAPS correlations 

Actual situation 

.543 (p = .065) 

Objectives 

.377 (p = .159) 

Measures/strategies 

.820** (p = .003) 

Control 

.789** (p = .006) 

Overall score for a person’s ability to solve 
problems 

.807** (p = .004) 


("(In order actually to establish whether the ‘ability to solve problems' construct is being valid- 
ly ascertained, an external criterion would have to be applied. This has been difficult hith- 
erto because, to our knowledge, there have never been any proven, valid instruments that 
actually measure a person's ability to solve problems. Although there are a number of ques- 
tionnaires on the subject, to use these as an external criterion is problematical in that infor- 
mation provided by students themselves and measurements of skills do not necessarily have 
to be connected. 
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Consequences for teacher training and for 
subsequent investigations 

International comparative studies like PISA have shown that German 
students are often insufficiently capable of applying their knowledge and 
solving complex problems. With respect to a person’s ability to solve prob- 
lems ( 12 ) ascertained within the framework of research into interdisciplinary 
skills in the PISA 2003 study, it was revealed that although German stu- 
dents were above the OECD average, they were significantly behind the 
international leaders (Korea, Finland and Japan). Moreover, 14.1 percent 
of young people in Germany have to be classified below the first skill lev- 
el as far as problem-solving is concerned (cf. OECD, 2004). 

As mentioned at the beginning, the relevant literature contains plenty 
of methodical advice on the design of problem-based teaching and learn- 
ing arrangements that might address this deficiency. However, the prob- 
lem remains that teachers are often still very attached to conventional teach- 
ing methods and hence make insufficient use of the opportunities to increase 
a person’s ability to solve problems. To improve the situation, greater use 
of more innovative methods is required in teacher training and further ed- 
ucation events for teachers who are already established. 

However, this does not overcome the problem that, at present, there 
are no instruments for measuring a person's ability to solve problems avail- 
able that are suitable for use in vocational colleges. As most of what is learnt 
is that which is tested at the end of the year - and is at present still pre- 
dominantly factual knowledge - this lack of available test instruments and 
measurement methods may have a negative impact on the design and im- 
plementation of teaching and learning processes. 

MAPS is a first step towards closing this gap by developing a practical 
instrument. Findings so far provide starting points enabling development 
of a reliable and possibly also valid instrument for measuring a person’s abil- 
ity to solve problems. As the previous study was carried out in a universi- 
ty environment, though the instrument was designed for use at vocation- 
al colleges, the next study will be carried out at a vocational college. It should 
first be examined here whether students and teachers can work with this 
instrument, in particular whether the MAPS rating categories are appropri- 
ate. Secondly, it should be checked again with a larger sample how reli- 
ably MAPS measures the person's ability to solve problems. The design 
is structured as follows: 

In a first step, students’ ability to solve problems is ascertained by means 


( 12 ) What was tested was the ability to use cognitive processes in order to solve real, interdis- 
ciplinary problems in which the solution is not immediately obvious. The questions set three 
types of problem (making decisions, analysing and outlining systems and looking for errors) 
and related to requirements for life outside school (e.g. in leisure and professional situations), 
in which problem-based action is required. The international test involves paper and pen ques- 
tions, in particular skill at solving analytical problems. 
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of Staudel's skills questionnaire ( 13 ) (1987) in 5 classes (N = 100). Using 
these preliminary test values, the whole group is divided into two equal sub- 
groups that should be indistinguishable with respect to students’ ability to 
solve problems. This should prevent the findings being distorted by a sys- 
tematic ability effect (a person’s ability to solve problems) in one group ( 14 ). 

In a second step, the groups are given identical problems. As it cannot 
be expected that all classes in an educational subject area will have the same 
knowledge, a problem is chosen from students’ everyday life. This should 
prevent an uneven distribution of prior knowledge falsifying the results. One 
group tackles the open problem that is evaluated according to the AIT sys- 
tem. The other group is given the problem according to MAPS, which is linked 
with specific questions and evaluated according to the MAPS system. In 
both groups, the solutions are evaluated by to encoders each. 

Using this design - on the basis of the problem-solving of one subgroup 
-the MAPS interrater reliability can be rechecked. It is thereby possible 
to establish how effective the instrument is in a school environment. As 
there are three measures of a person's ability to solve problems in this pro- 
cedure, it is also possible to examine whether the instruments measure 
a similar or identical construct. For each instrument there are in this case 
two ‘external criteria’ which enable assessments to be made regarding va- 
lidity. ■ 


Bibliography 

Achtenhagen, F. Lebenslanges Lernen Im Beruf: Seine Grundlegung im 
Kindes- und Jugendalter. Opladen: Leske und Budrich, 2000. 

Bereiter, C.; Scardamalia, M. Intentional learning as a goal of instruction. 
In: Resnick, L.B. (ed.). Knowing , learning and instruction: Essays in 
honour of Robert Glaser. Hillsdale, NJ: Lawrence Erlbaum, 1989, p. 361 - 
392. 

Bohl, Th. Unterrlchtsmethoden In der Realschule. Eine emplrlsche Unter- 
suchung zum Gebrauch ausgewahlter Unterrichtsmethoden an 
staatlichen Realschulen In Baden-Wurttemberg. Ein Beitrag zur deskrip- 
tlven Unterrichtsforschung. Bad Heilbrunn/Obb. 2000. 

Bransford, J.D.; Stein, B.S. The IDEAL Problem Solver (2 nd Edition). New 
York: W.H. Freeman, 1993. 

Buttler, F. Working landscape up to 2010. In: Achtenhagen, F.; John, E.G. 
(ed.). Mehrdlmenslonale Lehr-Lern-Arrangements. Wiesbaden: Gabler, 
1992, p. 162-182. 


( 13 ) The subscales adequate problem-solving behaviour' and ‘heuristic skill’ are used here to meas- 
ure a person’s ability to solve problems irrespective of the subject. 

( 14 ) In principle, it would also be possible to form the groups on the basis of their demonstrated 
ability to solve problems (measured by AIT or MAPS). However, this procedure is very ex- 
pensive and recognition effects could distort the subsequent findings. 



European journal of vocational training 

100 No 41 -2007/2 


Cognition And Technology Group At Vanderbilt. The Jasper Series as an 
example of anchored instruction: Theory, program description, and as- 
sessment data. In: Educational Psychologist, 1992, 27 (3), p. 291-315. 

Dichanz, H.; Schwittmann, D. Everyday educational methods. Investigation 
of teaching method combinations and models. In: Die deutsche 
Schule, 1 986, 78, 3, p. 327-337. 

Dirksmeier, C. Erfassung von Problemlosefahigkeit. Konstruktion und er- 
ste Validierung eines Diagnostischen Inventars. Munster, New York: Wax- 
mann, 1991. 

Dohmen, G. ‘Self-organised learning’ as a starting point for a required new 
start in further education. In: Federal Ministry for Education and Research 
(ed.). Selbstgesteuertes Lernen, Dokumentation zum KAW-KongreB vom 
4. bis 6. November, 1998 in Konigswinter. Bonn, 1999, p. 27-32. 

Dorner, D. Problemlosen als Informationsverarbeitung. (Kohlhammer) 
Stuttgart, 1976. 

Dorner, D. Cognitive processes and the organisation of action. In: Hack- 
er, W.; Volpert, W.; von Cranach, M. (ed.). Kognitive und motivationale 
Aspekte der Handlung. (Huber) Berne, Stuttgart, Vienna, 1983, p. 26- 
37. 

Dorner, D. Bauplan fureine Seele. Reinbek bei Hamburg: Rohwohlt, 1999. 

Hage, K.; Bischoff, H.; Dichanz, H.; Eubel, K.D.; Oehlschlager, H.J.; 
Schwittmann, D. Das Methodenrepertoire von Lehrern. Eine Un- 
tersuchung zum Schulalltag der Sekundarstufe I. Opladen: Leske 
+ Budrich, 1985. 

Hussy, W.; Seeling, D. Tests for a person's ability to solve sector-specific 
problems. In: Wirtschaftspsychologie aktuell, 2004a, volume 2004, is- 
sue 2, p. 10-14. 

Hussy, W.; Seeling, D. Tests zur Erfassung von branchenspezifischer und 
genereller Problemlosefahigkeit. Testbeschreibung. 2004b 
Downloadable from the Internet at the following address: http://www.per- 
sonal-point.de/personaldiagnostik/schriftliche_tests/info/personalpoint_ 
Tests.pdf (status: on 01.08.2005). 

Jonassen, D.H. Learning to Solve Problems. An Instructional Design Guide. 
San Francisco, CA.: Pfeiffer, 2004. 

Kessler, W. Companies under pressure of globalization. In: Informationen 
zur politischen Bildung (issue 280) 2003. Downloadable from the In- 
ternet at the following address: http://www.bpb.de/publikationen/ 
SN187N.html. (status: 26.01.2007). 

Kommers, P.; Jonassen, D.H.; Mayes T. (ed.) Cognitive tools for learning. 
Heidelberg FRG: Springer-Verlag, 1992. 

Miller, G.; Galanter, E.; Pribram, K. Plans and the structure of behavior. New 
York (Henry Holt and Company), 1960. 

OECD. Problem Solving for tomorrow’s world - First measures of cross- 
curricular skills from PISA 2003. Paris: OECD, 2004. 

Oser, F.; Spychiger, M.; Mahler, F.; Reber, S. Do people learn from their 



Developing an instrument for identifying a person’s ability to solve problems 

Results of a pilot study 

Eveline Wuttke, Karsten D. Wolf 


mistakes? Regarding the development of a mistakes culture in school. 
Concluding report to the Swiss National Fund for the Promotion of the 
Scientific Research. Freiburg: Department of Educational Sciences at 
Freiburg University/Switzerland, 2002. 

Patzold, G.; Klusmeyer, J.; Wingels, J.; Lang, M. Lehr-Lern-Methoden in 
der beruflichen Bildung. Eine empirische Untersuchung in ausgewahlten 
Berufsfeldern. Oldenburg University: Oldenburg, 2003. 

Picot, A. The transformation of business in an information society. In: Frank- 
furter Allgemeine Zeitung, 2000, No. 46, p. 29. 

Reetz, L. Regarding the relationship between key qualifications, skills and 
education. In: Tramm, T.; Sembill, D.; Klauser, F.; John, E.G. (ed.). Pro- 
fessionalisierung kaufmannischer Berufsbildung: Beitrage zur Off- 
nung der Wirtschaftspadagogik fur die Anforderungen des 21. Jahrhun- 
derts. Festschrift zum 60. Geburtstag von Frank Achtenhagen. Frank- 
furt am Main, Berlin, Berne, Brussels, New York: Peter Lang, 1999, p. 
32-51. 

Santjer-Schnabel, I. Emotional sensitivity in a learning environment open 
to independent organisation. Considerations for taking further account 
of physiological aspects. Hamburg: Dr Kovac, 2002. 

Schumacher, L. Emotional sensitivity and motives in learning groups. 
Hamburg: Dr Kovac, 2002. 

Schunck, A. Subjektive Theorien von Berufsfachschulern zu einem plan- 
spielgestutzten Betriebswirtschaftslehre-Unterricht. Dissertation zur 
Erlangung des wirtschaftswissenschaftlichen Doktorgrades des Fach- 
bereichs Wirtschaftswissenschaften der Universitat Gottingen 1 993, vol- 
ume 19. 

Seifried, J. Subject-didactic Variations in a learning environment open to 
independent organisation - An empirical study into accountancy train- 
ing. Wiesbaden: Deutscher Universitats-Verlag, 2004. 

Sembill, D. Problemldsefahigkeit, Handlungskompetenz und emotionale 
Befindlichkeit. Gottingen: Hogrefel, p. 992a. 

Sembill, D. Handlungs- und Emotionsstrukturen. Operationalisierungen, 
Rekonstruktionen und Ergebnisse. Dokumentationsband zur Monogra- 
phie: Problemldsefahigkeit, Handlungskompetenz und Emotionale 
Befindlichkeit. ZielgroBen Forschenden Lernens. Gottingen/Mannheim, 
1992b. 

Sembill, D. The will to have an option - Creative power in a tension 
relationship between innovation and organisation. In: Bunk, 
G.P.; Lassahn, R. (Ed). Festschrift fur Artur Fischer zum 75. Geburtstag. 
(Ehgart & Albohn) Steinbach bei GieRen, 1995, p. 125-146. 

Sembill, D. Abschlussbericht AZ. Se 573/4-2 to the German Research Coun- 
cil within the framework of the programme focusing on ‘Teaching and 
learning processes in initial business education’, 2004. 
Downloadable from the Internet at the following address: http://wipaed. 
sowi.uni-bamberg.de/download-pdf/DFG_bericht_sole2.pdf (status: 
14.09.2005). 



European journal of vocational training 

102 No 41 -2007/2 


Sembill, D.; Wolf, K.D. Praxisreader Setbstorganisiertes Lernen. Wiesbaden: 
Hesse State Institute for Education (in publication), 2000. 

Sembill, D. Wolf, K.D.; Wuttke, E.; Schumacher, L. Self-organized Learn- 
ing in Vocational Education. Foundation, Implementation, and Evalu- 
ation. In: Beck, K. (Ed.). Teaching-Learning Processes in Vocational Ed- 
ucation. Foundations of Modern Training Programmes. Frankfurt et al.: 
Peter Lang, 2002, p. 267-295. 

Spychiger, M. Mistakes as a window to the learning process: Regarding de- 
velopment of a mistakes culture in practical education. In: Journal fur 
Lehrerlnnenbildung, 2003, 3, 2, p. 31-38. 

Spychiger, M.; Gut, K.; Rohrbach, D.; Oser, F. Mistakes in everyday life. 
Experience to learn from mistakes. Schriftenreihe zum Projekt ‘Lernen 
Menschen aus Fehlern?’, No. 7. Freiburg/Switzerland: Department of 
Educational Sciences of Freiburg University, Switzerland, 1999. 

Staudel, T. Problemlosen , Emotionen und Kompetenz. Die Uberprufung eines 
integrativen Konstrukts. Regensburg: Roderer, 1987. 

Whitehead, A.N. The Aims of Education & Other Essays. New York: Macmil- 
lan, 1929. 

Wolf, K.D. Gestaltung und Einsatz einer internetbasierten Lernumgebung 
zur Unterstutzung Selbstorganisierten Lernens. Hamburg: Dr Kovac, 
2003. 

Wuttke, E. Motivation und Lernstrategien in einer selbstorganisationsoffe- 
nen Lernumgebung. Eine empirische Untersuchung bei Industriekau- 
fleuten. Dissertation. Frankfurt et al.: (Peter Lang), 1999. 

Wuttke, E. Unterrichtskommunikation und Wissenserwerb. Zum Einfluss von 
Kommunikation aufden Prozess der Wissensgenerierung. Postdoctor- 
al thesis. Frankfurt: Peter Lang, 2005. 



